Papers with multi-modal pre-training model
TableVLM: Multi-modal Pre-training for Table Structure Recognition (2023.acl-long)
Copied to clipboard
| Challenge: | Tables are useful for displaying data in an organized manner, but they are difficult to extract from images because of their structure, notation, and representation. |
| Approach: | They propose a multi-modal pre-training model for table structure recognition that captures table structure-related features by multiple unsupervised objectives inspired by masked visual-language modeling. |
| Outcome: | The proposed model improves tree-editing-distance-score on ComplexTable by 1.97% . |
LayoutMask: Enhance Text-Layout Interaction in Multi-modal Pre-training for Document Understanding (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained models on document images with transformer-based backbones have led to significant performance gains in this field. |
| Approach: | They propose a multi-modal pre-training model that combines text, layout and image . they propose to use local 1D position instead of global 1D positions as layout input . |
| Outcome: | The proposed model can achieve state-of-the-art results on a wide variety of VrDU problems. |